Issue

https://jsw.ibm.com/browse/MASCORE-13109

Description

This PR implements a comprehensive Redis-based distributed locking mechanism to replace the legacy Git branch-based locking system for GitOps operations. The change addresses critical issues with concurrent pipeline executions that previously caused race conditions, merge conflicts, and branch pollution.

Key Changes

1. Redis Locking Infrastructure (image/cli/mascli/functions/gitops_utils)

• Added Redis connection management with TLS support
• Implemented atomic lock acquisition using Redis SET NX operations
• Added lock release with ownership verification using Lua scripts
• Included automatic lock expiry (TTL) to prevent orphaned locks
• Added comprehensive retry logic with configurable parameters
• Default behavior: GITOPS_USE_REDIS_LOCKING=false (Git branch locking) to avoid breaking changes
• Opt-in Redis: Set GITOPS_USE_REDIS_LOCKING=true to enable Redis locking (fails if Redis unavailable)

2. Container Image Updates

• Added redis-cli installation script (image/cli/install/install-redis-cli.sh)
• Updated Dockerfile to include redis-cli in the CLI container image (v19.6.1+)
• Supports multiple package managers (microdnf, dnf, yum, apt-get, apk)

3. Migrated GitOps Functions (10 functions)

All functions now support gitops_lock_and_modify with Redis locking (opt-in):

• gitops_cp4d_service
• gitops_suite_app_config
• gitops_deprovision_app_config
• gitops_mas_config
• gitops_suite_workspace
• gitops_deprovision_suite_workspace
• gitops_deprovision_cp4d_service
• gitops_db2u_database
• gitops_rds_db2_database
• gitops_deprovision_db2u_database

4. Documentation

• Comprehensive setup guide (docs/redis-locking-setup.md)
• IBM Toolchain configuration script (docs/ibm-toolchain-redis-setup.sh)
• Migration examples and troubleshooting guide

Impact

Reliability Improvements (when Redis enabled):

• Eliminates race conditions through atomic Redis operations
• Prevents merge conflicts in concurrent updates
• Removes temporary Git branch pollution
• Automatic lock expiry prevents orphaned locks
• Explicit control: fails immediately if Redis is unavailable when enabled

Non-Breaking Change:

• Default: GITOPS_USE_REDIS_LOCKING=false uses existing Git branch locking
• Opt-in: Set GITOPS_USE_REDIS_LOCKING=true to enable Redis locking
• Redis is optional - existing deployments continue to work without changes
• When Redis enabled, it is required (no fallback to Git)
• Operations fail immediately if Redis is unavailable when explicitly enabled

Test Results

Manual Testing

• ✅ Redis connection validation with TLS
• ✅ Lock acquisition and release operations
• ✅ Concurrent pipeline execution without conflicts
• ✅ Lock TTL expiry behavior
• ✅ Retry logic under contention
• ✅ All 10 migrated GitOps functions tested
• ✅ Default behavior (Git locking) unchanged
• ✅ Error handling when Redis enabled but unavailable

Integration Testing

• ✅ IBM Cloud Redis instance connectivity
• ✅ IBM Toolchain pipeline execution
• ✅ Multi-pipeline concurrent operations
• ✅ Error handling for Redis unavailability
• ✅ Backward compatibility with existing deployments

Configuration Testing

• ✅ TLS certificate handling
• ✅ Authentication (username/password)
• ✅ Environment variable configuration
• ✅ Setup script validation (ibm-toolchain-redis-setup.sh)
• ✅ Default behavior without Redis configuration

Backporting

Related Pull Requests

https://github.ibm.com/maximoappsuite/saas-deploy-py/pull/262

⚠️ Notes for Reviewers

• Ensure you have understood the guidelines before proceeding with a review.
• Ensure all sections in the PR template are appropriately completed.
• Non-breaking change: Default behavior (GITOPS_USE_REDIS_LOCKING=false) maintains existing Git branch locking
• Opt-in feature: Redis locking requires explicit GITOPS_USE_REDIS_LOCKING=true
• Explicit control: When Redis enabled, operations fail if Redis unavailable (no silent fallback)
• Review the migration strategy in docs/redis-locking-setup.md
• Verify the setup script docs/ibm-toolchain-redis-setup.sh for IBM Cloud integration
• Check that all 10 migrated functions follow the new locking pattern consistently
• Validate error handling for Redis unavailability scenarios
• Review security aspects: TLS configuration, credential handling, and password masking
• Confirm backward compatibility for existing deployments without Redis